{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "da1c70c2",
   "metadata": {},
   "source": [
    "##### **** The pip install for the transform needs to be adapted to use the appropriate release level. Alternatively, The venv running the jupyter lab could be pre-configured with a requirement file that includes the right release. Example for transform developers working from git clone:\n",
    "```\n",
    "make venv \n",
    "source venv/bin/activate \n",
    "pip install jupyterlab\n",
    "venv/bin/jupyter lab\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0fbe6e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "## This is here as a reference only\n",
    "# Users and application developers must use the right tag for the latest from pypi\n",
    "%pip install 'data-prep-toolkit-transforms[ml_filter]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42fe832b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7b446ec-3dff-4a06-ba49-d22eb0ed165f",
   "metadata": {},
   "source": [
    "##### 1 • Import the transform runtime and utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "915b4c2b-a4d8-469a-8b22-f26156bfd63a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "from dpk_ml_filter.runtime import MLFilter\n",
    "from dpk_ml_filter.transform import get_transform_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48dd5577-8e64-4bfa-a33b-c8b545cc3c9d",
   "metadata": {},
   "source": [
    "##### 2 • Required parameters for the transform\n",
    "| Name  | Default Value | Description |\n",
    "|------------|----------|--------------|\n",
    "| **lm_filter_config** | **_text_** | File name with the conditions for the filter, in YAML format. |\n",
    "| **lm_filter_lang_column_name** | **lang** | The column name with language identifier used in the filter conditions. |\n",
    "| **lm_filter_column_prefix** | _not set_ | A prefix for the names referenced in the conditions file.|\n",
    "| **lm_filter_ignore_missing_columns** | **False** | By default, the transform will fail if any for the values referenced in the conditions is missing. Set this parameter to ignore all the missing columns. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "556560ca-1596-4dc0-a246-99b0f2ade72b",
   "metadata": {},
   "source": [
    "##### 3 • Excerpt from a conditions file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39943e3-3406-4a7a-85df-446f021ec63c",
   "metadata": {},
   "source": [
    "```yaml\n",
    "# the default section is applied to all languages, along with any language specific condition\n",
    "default:\n",
    "    avg_word_length_min: 3.2\n",
    "    avg_word_length_max: 7.0\n",
    "\n",
    "# the laguage sections define conditions for each language separately\n",
    "\n",
    "en:\n",
    "    num_words_min: 110\n",
    "    lid_score_min: 0.5\n",
    "fr:\n",
    "    num_words_min: 100\n",
    "    lid_score_min: 0.6\n",
    "\n",
    "# by default a all enties with a language not specified here will be removed from the output, unless an <other> section is defined\n",
    "pt: \n",
    "\n",
    "# the other section is applied to all the languages without an explicit section\n",
    "other:\n",
    "    num_words_min: 1000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7234563c-2924-4150-8a31-4aec98c1bf33",
   "metadata": {},
   "source": [
    "##### 4 • Setup the transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "badafb96-64d2-4bb8-9f3e-b23713fd5c3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = MLFilter(\n",
    "    ml_filter_config=\"test-data/input/cleansing-config.yaml\",\n",
    "    ml_filter_lang_column_name=\"lang\",\n",
    "    ml_filter_column_prefix=\"e_\",\n",
    "    input_folder=\"test-data/input\", \n",
    "    output_folder=\"output\" \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "853aba4b-0094-4264-8677-5371f1141142",
   "metadata": {},
   "source": [
    "##### 5 • Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "930748f2-f7f5-459c-b2e7-8e3c54873838",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:04:51 INFO - Filtering parameters are: {\"column_prefix\": \"e_\", \"lang_column_name\": \"lang\", \"config\": \"test-data/input/cleansing-config.yaml\", \"ignore_missing_columns\": false}\n",
      "13:04:51 INFO - pipeline id pipeline_id\n",
      "13:04:51 INFO - code location None\n",
      "13:04:51 INFO - data factory data_ is using local data access: input_folder - test-data/input output_folder - output\n",
      "13:04:51 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:04:51 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:04:51 INFO - orchestrator ml_filter started at 2025-03-12 13:04:51\n",
      "13:04:51 INFO - Number of files is 2, source profile {'max_file_size': 0.24041080474853516, 'min_file_size': 0.21344757080078125, 'total_file_size': 0.4538583755493164}\n",
      "13:04:51 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "13:04:51 INFO - Completed 2 files (100.0%) in 0.0 min\n",
      "13:04:51 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "13:04:51 INFO - done flushing in 0.0 sec\n",
      "13:04:51 INFO - Completed execution in 0.001 min, execution result 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transform.transform()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3df5adf-4717-4a03-864d-9151cd3f134b",
   "metadata": {},
   "source": [
    "##### 6 • Show the number of rows before and after applying the transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "845a75cf-f4a9-467d-87fa-ccbac1c9beb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before: 100, after: 83\n"
     ]
    }
   ],
   "source": [
    "before = len(pandas.read_parquet('test-data/input/1.parquet', engine='pyarrow'))\n",
    "after = len(pandas.read_parquet('output/1.parquet', engine='pyarrow'))\n",
    "print(f\"before: {before}, after: {after}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aef6ac9-96cf-40ad-a472-b5d9036436e5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
